Deep learning & remote sensing : pushing the frontiers in image segmentation

Dissertação (Mestrado em Informática) — Universidade de Brasília, Instituto de Ciências Exatas, Departamento de Ciência da Computação, Brasília, 2022.A segmentação de imagens visa simplificar o entendimento de imagens digitais e métodos de aprendizado profundo usando redes neurais convolucionais permitem a exploração de diferentes tarefas (e.g., segmentação semântica, instância e panóptica). A segmentação semântica atribui uma classe a cada pixel em uma imagem, a segmentação de instância classifica objetos a nível de pixel com um identificador exclusivo para cada alvo e a segmentação panóptica combina instâncias com diferentes planos de fundo. Os dados de sensoriamento remoto são muito adequados para desenvolver novos algoritmos. No entanto, algumas particularidades impedem que o sensoriamento remoto com imagens orbitais e aéreas cresça quando comparado às imagens tradicionais (e.g., fotos de celulares): (1) as imagens são muito extensas, (2) apresenta características diferentes (e.g., número de canais e formato de imagem), (3) um grande número de etapas de préprocessamento e pós-processamento (e.g., extração de quadros e classificação de cenas grandes) e (4) os softwares para rotulagem e treinamento de modelos não são compatíveis. Esta dissertação visa avançar nas três principais categorias de segmentação de imagens. Dentro do domínio de segmentação de instâncias, propusemos três experimentos. Primeiro, aprimoramos a abordagem de segmentação de instância baseada em caixa para classificar cenas grandes. Em segundo lugar, criamos um método sem caixas delimitadoras para alcançar resultados de segmentação de instâncias usando modelos de segmentação semântica em um cenário com objetos esparsos. Terceiro, aprimoramos o método anterior para cenas aglomeradas e desenvolvemos o primeiro estudo considerando aprendizado semissupervisionado usando sensoriamento remoto e dados GIS. Em seguida, no domínio da segmentação panóptica, apresentamos o primeiro conjunto de dados de segmentação panóptica de sensoriamento remoto e dispomos de uma metodologia para conversão de dados GIS no formato COCO. Como nosso primeiro estudo considerou imagens RGB, estendemos essa abordagem para dados multiespectrais. Por fim, melhoramos o método box-free inicialmente projetado para segmentação de instâncias para a tarefa de segmentação panóptica. Esta dissertação analisou vários métodos de segmentação e tipos de imagens, e as soluções desenvolvidas permitem a exploração de novas tarefas , a simplificação da rotulagem de dados e uma forma simplificada de obter previsões de instância e panópticas usando modelos simples de segmentação semântica.Image segmentation aims to simplify the understanding of digital images. Deep learning-based methods using convolutional neural networks have been game-changing, allowing the exploration of different tasks (e.g., semantic, instance, and panoptic segmentation). Semantic segmentation assigns a class to every pixel in an image, instance segmentation classifies objects at a pixel level with a unique identifier for each target, and panoptic segmentation combines instancelevel predictions with different backgrounds. Remote sensing data largely benefits from those methods, being very suitable for developing new DL algorithms and creating solutions using top-view images. However, some peculiarities prevent remote sensing using orbital and aerial imagery from growing when compared to traditional ground-level images (e.g., camera photos): (1) The images are extensive, (2) it presents different characteristics (e.g., number of channels and image format), (3) a high number of pre-processes and post-processes steps (e.g., extracting patches and classifying large scenes), and (4) most open software for labeling and deep learning applications are not friendly to remote sensing due to the aforementioned reasons. This dissertation aimed to improve all three main categories of image segmentation. Within the instance segmentation domain, we proposed three experiments. First, we enhanced the box-based instance segmentation approach for classifying large scenes, allowing practical pipelines to be implemented. Second, we created a bounding-box free method to reach instance segmentation results by using semantic segmentation models in a scenario with sparse objects. Third, we improved the previous method for crowded scenes and developed the first study considering semi-supervised learning using remote sensing and GIS data. Subsequently, in the panoptic segmentation domain, we presented the first remote sensing panoptic segmentation dataset containing fourteen classes and disposed of software and methodology for converting GIS data into the panoptic segmentation format. Since our first study considered RGB images, we extended our approach to multispectral data. Finally, we leveraged the box-free method initially designed for instance segmentation to the panoptic segmentation task. This dissertation analyzed various segmentation methods and image types, and the developed solutions enable the exploration of new tasks (such as panoptic segmentation), the simplification of labeling data (using the proposed semi-supervised learning procedure), and a simplified way to obtain instance and panoptic predictions using simple semantic segmentation models

Automatic detection of roads and pivot tracks using high spatial resolution images from PRISM-ALOS sensor

Este artigo apresenta um novo método para a detecção de estradas e trilhas de pivôs em imagens pancromáticas de alta resolução. A abordagem é dividida em três módulos principais: primeiro, uma filtragem espacial é aplicada a imagem para a extração de magnitudes e direções de descontinuidades, então uma quantização em 5 níveis é aplicada às magnitudes e um algoritmo de consistência hierárquica é desenvolvido para checar e unir cada janela a sua vizinhança, priorizando direções com os maiores e mais consistentes valores de magnitude; segundo, as hipóteses mais fortes são mantidas enquanto as mais fracas são retiradas da imagem, com o intuito de eliminar partes desconectadas; e finalmente, a transformada de Hough é aplicada nas hipóteses para completar as estradas detectadas e também identificar possíveis trilhas de pivôs entre elas. Este método foi aplicado em imagens do satélite ALOS em d iferentes áreas do oeste do estado da Bahia, Brasil, e os resultados são fornecidos aqui. _________________________________________________________________________________ ABSTRACTThis article introduces a new method for road and pivot tracks detection for high resolution pancromatic satellite images. The approach is divided on three main modules: firstly, a spatial derivative mask is applied to the image for extraction of magnitudes and directions of discontinuities, then a five level quantization is applied on the magnitudes and a hierarchical consistency algorithm is developed by checking and linking each window to its neighborhood prioritizing directions with higher and consistent magnitude values; secondly, the road stronger hypotheses are left while the weaker ones are cleaned from the images, with the purpose to eliminate disconnected pieces; and finally, a Hough transformation is applied on the road hypotheses in order to complete the roads detected and to also identify possible pivot tracks among them. The method was applied to images of the ALOS satellite of different areas of the west of state of Bahia, Brazil, and results are provided here

Bounding Box-Free Instance Segmentation Using Semi-Supervised Learning for Generating a City-Scale Vehicle Dataset

Vehicle classification is a hot computer vision topic, with studies ranging from ground-view up to top-view imagery. In remote sensing, the usage of top-view images allows for understanding city patterns, vehicle concentration, traffic management, and others. However, there are some difficulties when aiming for pixel-wise classification: (a) most vehicle classification studies use object detection methods, and most publicly available datasets are designed for this task, (b) creating instance segmentation datasets is laborious, and (c) traditional instance segmentation methods underperform on this task since the objects are small. Thus, the present research objectives are: (1) propose a novel semi-supervised iterative learning approach using GIS software, (2) propose a box-free instance segmentation approach, and (3) provide a city-scale vehicle dataset. The iterative learning procedure considered: (1) label a small number of vehicles, (2) train on those samples, (3) use the model to classify the entire image, (4) convert the image prediction into a polygon shapefile, (5) correct some areas with errors and include them in the training data, and (6) repeat until results are satisfactory. To separate instances, we considered vehicle interior and vehicle borders, and the DL model was the U-net with the Efficient-net-B7 backbone. When removing the borders, the vehicle interior becomes isolated, allowing for unique object identification. To recover the deleted 1-pixel borders, we proposed a simple method to expand each prediction. The results show better pixel-wise metrics when compared to the Mask-RCNN (82% against 67% in IoU). On per-object analysis, the overall accuracy, precision, and recall were greater than 90%. This pipeline applies to any remote sensing target, being very efficient for segmentation and generating datasets.Comment: 38 pages, 10 figures, submitted to journa

Comparação dos métodos green e atrem para correção atmosférica de imagens hiperespectrais aviris

O presente trabalho tem como propósito realizar uma análise comparativados métodos Green e ATREM para a correção atmosférica de imagens do sensor hiperespectral AVIRIS. Também foi avaliada a aplicação do método complementar EFFORT, que proporciona uma filtragem de eventuais resíduos atmosféricos verificadosapós a correção. Verificou-se que o método Green apresentou melhores resultados emcomparação ao método ATREM. O emprego do EFFORT permitiu uma melhora dosespectros do ATREM, porém apresentou apenas uma melhora moderada sobre os resultados do método Green. _______________________________________________________________________________ ABSTRACTThe present work deals with a comparative analysis of two atmospheric correction methods for AVIRIS data: the Green and the ATREM methods. It is also analysed the procedure EFFORT, associated to the atmospheric correction, which is applied after the correction in order to eventually filter atmospheric residues. It is shown that the Green’s Method presented better results, compared to the ATREM Method. The use of the EFFORT procedure produced a significant improvement on the ATREM’s results, but the improvement on the Green’s Method results was moderate

Panoptic Segmentation Meets Remote Sensing

Panoptic segmentation combines instance and semantic predictions, allowing the detection of "things" and "stuff" simultaneously. Effectively approaching panoptic segmentation in remotely sensed data can be auspicious in many challenging problems since it allows continuous mapping and specific target counting. Several difficulties have prevented the growth of this task in remote sensing: (a) most algorithms are designed for traditional images, (b) image labelling must encompass "things" and "stuff" classes, and (c) the annotation format is complex. Thus, aiming to solve and increase the operability of panoptic segmentation in remote sensing, this study has five objectives: (1) create a novel data preparation pipeline for panoptic segmentation, (2) propose an annotation conversion software to generate panoptic annotations; (3) propose a novel dataset on urban areas, (4) modify the Detectron2 for the task, and (5) evaluate difficulties of this task in the urban setting. We used an aerial image with a 0,24-meter spatial resolution considering 14 classes. Our pipeline considers three image inputs, and the proposed software uses point shapefiles for creating samples in the COCO format. Our study generated 3,400 samples with 512x512 pixel dimensions. We used the Panoptic-FPN with two backbones (ResNet-50 and ResNet-101), and the model evaluation considered semantic instance and panoptic metrics. We obtained 93.9, 47.7, and 64.9 for the mean IoU, box AP, and PQ. Our study presents the first effective pipeline for panoptic segmentation and an extensive database for other researchers to use and deal with other data or related problems requiring a thorough scene understanding.Comment: 40 pages, 10 figures, submitted to journa

COMPARAÇÃO DOS MÉTODOS GREEN E ATREM PARA CORREÇÃO ATMOSFÉRICA DE IMAGENS HIPERESPECTRAIS AVIRIS

The present work deals with a comparative analysis of two atmospheric correction methods for AVIRIS data: the Green and the ATREM methods. It is also analysed the procedure EFFORT, associated to the atmospheric correction, which is applied after the correction in order to eventually filter atmospheric residues. It is shown that the Green’s Method presented better results, compared to the ATREM Method. The use of the EFFORT procedure produced a significant improvement on the ATREM’s results, but the improvement on the Green’s Method results was moderate.O presente trabalho tem como propósito realizar uma análise comparativa dos métodos Green e ATREM para a correção atmosférica de imagens do sensor hiperespectral AVIRIS. Também foi avaliada a aplicação do método complementar EFFORT, que proporciona uma filtragem de eventuais resíduos atmosféricos verificados após a correção. Verificou-se que o método Green apresentou melhores resultados em comparação ao método ATREM. O emprego do EFFORT permitiu uma melhora dos espectros do ATREM, porém apresentou apenas uma melhora moderada sobre os resultados do método Green

Rice crop detection using LSTM, Bi-LSTM, and machine learning models from sentinel-1 time series

The Synthetic Aperture Radar (SAR) time series allows describing the rice phenological cycle by the backscattering time signature. Therefore, the advent of the Copernicus Sentinel-1 program expands studies of radar data (C-band) for rice monitoring at regional scales, due to the high temporal resolution and free data distribution. Recurrent Neural Network (RNN) model has reached state-of-the-art in the pattern recognition of time-sequenced data, obtaining a significant advantage at crop classification on the remote sensing images. One of the most used approaches in the RNN model is the Long Short-Term Memory (LSTM) model and its improvements, such as Bidirectional LSTM (Bi-LSTM). Bi-LSTM models are more effective as their output depends on the previous and the next segment, in contrast to the unidirectional LSTM models. The present research aims to map rice crops from Sentinel-1 time series (band C) using LSTM and Bi-LSTM models in West Rio Grande do Sul (Brazil). We compared the results with traditional Machine Learning techniques: Support Vector Machines (SVM), Random Forest (RF), k-Nearest Neighbors (k-NN), and Normal Bayes (NB). The developed methodology can be subdivided into the following steps: (a) acquisition of the Sentinel time series over two years; (b) data pre-processing and minimizing noise from 3D spatial-temporal filters and smoothing with Savitzky-Golay filter; (c) time series classification procedures; (d) accuracy analysis and comparison among the methods. The results show high overall accuracy and Kappa (>97% for all methods and metrics). Bi-LSTM was the best model, presenting statistical differences in the McNemar test with a significance of 0.05. However, LSTM and Traditional Machine Learning models also achieved high accuracy values. The study establishes an adequate methodology for mapping the rice crops in West Rio Grande do Sul

Remote sensing for monitoring photovoltaic solar plants in Brazil using deep semantic segmentation

Brazil is a tropical country with continental dimensions and abundant solar resources that are still underutilized. However, solar energy is one of the most promising renewable sources in the country. The proper inspection of Photovoltaic (PV) solar plants is an issue of great interest for the Brazilian territory’s energy management agency, and advances in computer vision and deep learning allow automatic, periodic, and low-cost monitoring. The present research aims to identify PV solar plants in Brazil using semantic segmentation and a mosaicking approach for large image classification. We compared four architectures (U-net, DeepLabv3+, Pyramid Scene Parsing Network, and Feature Pyramid Network) with four backbones (Efficient-net-b0, Efficient-net-b7, ResNet-50, and ResNet-101). For mosaicking, we evaluated a sliding window with overlapping pixels using different stride values (8, 16, 32, 64, 128, and 256). We found that: (1) the models presented similar results, showing that the most relevant approach is to acquire high-quality labels rather than models in many scenarios; (2) U-net presented slightly better metrics, and the best configuration was U-net with the Efficient-net-b7 encoder (98% overall accuracy, 91% IoU, and 95% F-score); (3) mosaicking progressively increases results (precision-recall and receiver operating characteristic area under the curve) when decreasing the stride value, at the cost of a higher computational cost. The high trends of solar energy growth in Brazil require rapid mapping, and the proposed study provides a promising approach

A Data-Centric Approach for Wind Plant Instance-Level Segmentation Using Semantic Segmentation and GIS

Wind energy is one of Brazil’s most promising energy sources, and the rapid growth of wind plants has increased the need for accurate and efficient inspection methods. The current onsite visits, which are laborious and costly, have become unsustainable due to the sheer scale of wind plants across the country. This study proposes a novel data-centric approach integrating semantic segmentation and GIS to obtain instance-level predictions of wind plants by using free orbital satellite images. Additionally, we introduce a new annotation pattern, which includes wind turbines and their shadows, leading to a larger object size. The elaboration of data collection used the panchromatic band of the China–Brazil Earth Resources Satellite (CBERS) 4A, with a 2-m spatial resolution, comprising 21 CBERS 4A scenes and more than 5000 wind plants annotated manually. This database has 5021 patches, each with 128 × 128 spatial dimensions. The deep learning model comparison involved evaluating six architectures and three backbones, totaling 15 models. The sliding windows approach allowed us to classify large areas, considering different pass values to obtain a balance between performance and computational time. The main results from this study include: (1) the LinkNet architecture with the Efficient-Net-B7 backbone was the best model, achieving an intersection over union score of 71%; (2) the use of smaller stride values improves the recognition process of large areas but increases computational power, and (3) the conversion of raster to polygon in GIS platforms leads to highly accurate instance-level predictions. This entire pipeline can be easily applied for mapping wind plants in Brazil and be expanded to other regions worldwide. With this approach, we aim to provide a cost-effective and efficient solution for inspecting and monitoring wind plants, contributing to the sustainability of the wind energy sector in Brazil and beyond

Rethinking panoptic segmentation in remote sensing : a hybrid approach using semantic segmentation and non-learning methods

This letter proposes a novel method to obtain panoptic predictions by extending the semantic segmentation task with a few non-learning image processing steps, presenting the following benefits: 1) annotations do not require a specific format [e.g., common objects in context (COCO)]; 2) fewer parameters (e.g., single loss function and no need for object detection parameters); and 3) a more straightforward sliding windows implementation for large image classification (still unexplored for panoptic segmentation). Semantic segmentation models do not individualize touching objects, as their predictions can merge; i.e., a single polygon represents many targets. Our method overcomes this problem by isolating the objects using borders on the polygons that may merge. The data preparation requires generating a one-pixel border, and for unique object identification, we create a list with the isolated polygons, attribute a different value to each one, and use the expanding border (EB) algorithm for those with borders. Although any semantic segmentation model applies, we used the U-Net with three backbones (EfficientNet-B5, EfficientNet-B3, and EfficientNet-B0). The results show that the following hold: 1) the EfficientNet-B5 had the best results with 70% mean intersection over union (mIoU); 2) the EB algorithm presented better results for better models; 3) the panoptic metrics show a high capability of identifying things and stuff with 65 panoptic quality (PQ); and 4) the sliding windows on a 2560×2560 -pixel area has shown promising results, in which the ratio of merged objects by correct predictions was lower than 1% for all classes.Instituto de Ciências Exatas (IE)Departamento de Ciência da Computação (IE CIC)Instituto de Ciências Humanas (ICH)Departamento de Geografia (ICH GEA